Learning to Hire Teams

Crowdsourcing and human computation are being employed in sophisticated projects that require the solution of a heterogeneous set of tasks. We explore the challenge of composing or hiring an effective team from an available pool of applicants for performing tasks required for such projects on an ongoing basis. How can one optimally spend budget to learn the expertise of workers as part of recruiting a team? How can one exploit the similarities among tasks as well as underlying social ties or commonalities among the workers for faster learning? We tackle these decision-theoretic challenges by casting them as an instance of online learning for best action selection with side-observations. We present algorithms with PAC bounds on the required budget to hire a near-optimal team with high confidence. We evaluate our methodology on simulated problem instances using crowdsourcing data collected from the Upwork platform. Introduction The success of a project or a collaborative venture depends critically on acquiring a team of contributors. Beyond increased performance and productivity, hiring a strong team can be important for effective collaboration, enhanced engagement, and increased retention of workers. “A small team of A+ players can run circles around a giant team of B and C players.” – Steve Jobs Crowdsourcing via online marketplaces further underscores the promise of developing procedures for identifying potential contributors and composing teams, even when a job requester and workers may be half a world apart. To date, online crowdsourcing markets have largely focused on micro-tasking through enlisting non-expert crowd of workers who work independently on simple tasks such as performing image annotation. With the increasing complexity of tasks that are crowdsourced, as well as enterprises outsourcing their work, the need to hire skilled workers with an eye to considerations of complementarity and coordinative efforts in a collaboration around problem solving is becoming important. Contract-based crowdsourcing is another emerging paradigm where workers are recruited on a contract for performing tasks on an ongoing basis. Online Copyright © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Side-observation graph for workers